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CHAPTER I 


THE TWO APPROACHES—A GENERAL STATEMENT 
OF THE PROBLEM! 


A. INTRODUCTION 


HE NUMEROUS investigations of the 
T validity of the Rorschach have been 
reviewed very thoroughly in three ar- 
ticles by Hertz (5, 6, 7), and the validity 
of the TAT has been similarly sum- 
marized by Tomkins (19). However, a 
brief restatement of the methods used 
by various investigators in establishing 
the validity of projective techniques may 
serve as an orientation to the particular 
questions considered in the present study. 
Thus far, the authorities who have re- 
viewed this problem have been con- 
cerned primarily with the types of evi- 
dence used for the validation of projec- 
tive techniques. Hertz (5) distinguished 
four main types of validation studies, as 
providing different kinds of evidence: 
(a) clinical studies, in which the useful- 
ness of these techniques is illustrated in 
the analysis of case histories, therapy, 
etc.; (b) experimental studies, in which 
changes in the test results are shown to 
accompany controlled changes in the in- 
dividual’s pattern of behavior; (c) studies 
of defined groups, in which certain pat- 
terns of test results are established as as- 
sociated with the characteristic behavior 
of known groups; and (d) predictive 
studies, in which the degree of agreement 
is measured between the description of 
personality derived from the results of 


‘This study was conducted under the auspices 
of the Veterans Administration Regional Office, 
San Francisco, while the author was in training 
in the Clinical Psychology Training Program of 
the Veterans Administration. The author is 
therefore oely indebted to the Veterans Ad- 
ministration for making this study possible. The 
Opinions stated in this report are, however, those 
of the author and do not necessarily reflect the 
viewpoint of the Veterans Administration. 


a projective technique and that obtained 
from an analysis of some criterion, for 
example, a life history. 

Although various writers (Tomkins 
[17], Macfarlane [12], and Symonds and 
Krugman [16]) have granted the pos- 
sibility of approaching the validation of 
projective techniques in various ways, 
they have, at the same time, been careful 
to emphasize that the chosen method 
must take into account the nature of the 
technique being validated, particularly 
the concept of personality underlying the 
use of this technique. The most compre- 
hensive argument concerning this point 
has been presented by Frank (3) in his 
classic discussion of the scientific basis of 
projective techniques. He pointed out 
that the “personality” which projective 
techniques are designed to evaluate is a 
framework of intervening concepts, a 
framework that relates the details of the 
individual’s manifest behavior in terms 
of a pattern of motivations and attitudes. 
Macfarlane (12) also has considered this 


use of interrelated constructs to be a - 


central problem “inherent in the valida- 
tion of projective techniques.” The point 
stressed by Frank is that personality as 
a configuration of functioning processes 
cannot be meaningfully broken up into 
isolated traits, but that part functions 


can only be described in terms of their 


interrelationships within the whole pat- 
tern. From this viewpoint, a description 
of personality derived from the results of 
a projective technique would require a 
method of validation which could test 
the accuracy of this description as a 
whole unit. 

This assumption concerning the rela- 
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derived from projective techniques, and 
the method employed for their valida- 
tion constituted the point of departure 
of the present research. In the light of 
this concept of personality, two divergent 
predictive approaches to validation were 
applied to an established projective tech- 
nique, the Rorschach. One of these ap- 
proaches, the matching method, was de- 
signed specifically to test the validity of 
description of personality as whole units. 
The other approach, which attempts to 
validate these descriptions item by item, 
does not necessarily take into account the 
Gestalt nature of these descriptions. The 
general intent of this study was to test 
the relative applicability of these two 
methods in investigating the validity of a 
projective technique. 


B. THe INTERPRETATION AS THE OBJECT 
OF VALIDATION 


Before proceeding with the description 
of these two methods, it may be well to 
emphasize that this study is concerned 
with methods of validating the interpre- 
tation or description of personality as 
derived from the responses of the sub- 
ject, rather than with consideration of 
specific scores or discrete responses. While 
there are merits in dealing with the so- 
called objective data of projective tests, 
this author agrees with such authorities 
as Hertz (6) and Macfarlane (12) that 
behind such scoring systems lie implicit 
assumptions about personality function- 
ing. It thus appears to this author more 
reasonable to deal directly with these 
interpretative assumptions and avoid the 
current controversies concerning scoring 


The question of whether or not a set of 
idiosyncratic responses represents in a 
rough manner the general pattern of 
functioning of the individual, and of 
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tionship between personality descriptions 


how this question may be answered, 
seems a legitimate object of study. 


C. THe MATCHING APPROACH 


It was clearly apparent to Vernon 
(19), even during the developmental 
stage of projective techniques, that there 
was a need for a statistical approach 
which would treat their interpretation as 
a single, whole unit. With this specific 
problem in mind, he developed what has 
become known in the literature as the 
matching method. As Vernon (18), Hun- 
ter (g), and Krugman (11) have used it, 
this method consists of the following 
procedures: An interpretative report 
from a projective technique and a case 
analysis are prepared, independently, for 
each individual in a given sample. The 
sample is divided into small groups, 
ranging from five to ten subjects, known 
as matching groups. The interpretative 
reports and case analyses of each group’ 
are presented, unidentified as to subject, 
to several judges who then attempi to 
match each of the test reports to the 
corresponding case analysis of the same 
individual’s life history. Validity is then 
expressed in terms of the success of this 
matching. Chapman (1) derived the sta- 
tistics for determining the chance varia- 
tion and the significance of the success 
of matching. Vernon (19) has added a 
formula for a coefficient of contingency, 
C, permitting a statement of the degree 
of relationship between the test and the 
criterion as implied in the success of the 
matching. 

Vernon (18) admitted that his method 
was “only a coarse beginning” to the 
validation of projective techniques and 


categories and their discrete meanings. / suggested two additional steps to this 


“tis 


? While most investigators have used matching 
groups consisting of an equal number of reports 
and case analyses, the matching groups may be 
uneven, e.g., ten interpretations to five analyses, 
or five to one. 
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procedure: (a) the homogeneity of the 
matching groups must be determined 
(obviously, a group of very similar re- 
ports would be more difficult to match 
than a group of very dissimilar reports); 
and (b) the reliability of the matching 
judges should be determined. 


The application of the matching method to 
the validation of projective techniques has pro- 
duced varying results. In his original study on 
the Rorschach, Vernon (18) reported an average 
contingency coefficient, C, of .833 + .0g15. Vernon 
noted that “the actual size of the C depends 
very largely on the degree of heterogeneity or 
distinctiveness of the subjects in each group. 
As far as possible, a normal degree of hetero- 
geneity was aimed at” by randomly selecting 
the cases for the groups out of the whole sample 
(18, p., 213) . However, Vernon’s matching groups 
may have been more heterogeneous than he 
assumed, as is suggested by the results of later 
studies. Hunter (g), in a study of fifty school 
children, reported that only five Rorschach re- 
ports were matched correctly by all four judges, 
and that each judge, singly, was successful in 
only go to 40 per cent of the matchings. She 
concluded, therefore, that matching was “of 
doubtful value” . . . “Calculated to differentiate 
only extreme cases” (g). Although this investiga- 
tor did not state her method of selecting the 
matching groups, she did remark that the per- 
sonality sketches were all very similar. 


The chief objection to the matching 
method, however, is that it permits, at 
best, only a very general statement about 
the accuracy of an interpretative report, 
namely, the statement that on the whole 
the report is similar to the personality 
pattern depicted by the criterion. While 
it is reassuring to know that an essential- 
ly meaningful interpretation may be 
drawn from a projective technique, it 
would be even more satisfactory to know 
how well the personality configuration is 
delineated in an interpretative report. 
If the interpretative report is, in gen- 
eral, similar to the case analysis, success- 
ful matching may occur, even though 
the accuracy of many of the statements 
within the report is dubious. In fact, 
Cronbach (2) considered that the match- 


ing method depends too much on the 
presence or absence of small clues. A pro- 
ponent of the matching method might 
argue that, if the same personality pat- 
tern is, in general, described in both the 
interpretative report and in the case 
analysis, then the statements about par- 
ticular functions of the personality would 
be likely to be similar in both descrip- 
tions. Unfortunately, the matching 
method does not provide any tests of this 
argument. 


D. AN ANALYysis METHOD 


In a study of the validity of the TAT, 
Harrison (4) introduced a_ procedure 
which simultaneously checked both the 
degree and area of the accuracy of his 
interpretations, His interpreters wrote 
out an “itemized analysis” of the test 
protocols, i.e., lists of separate interpreta- 
tive statements, and the case analyses 
were prepared in a similar fashion. The 
judges then compared the two sets of 
statements, item by item, denoting each 
item of the interpretation as “right,” 
“wrong,” or “?.” This index of validity 
was significantly higher for his sample of 
interpretations than for a random group 
of interpretations, or for a group of 
“mock” reports, matched randomly with 
the same criterion. Cronbach (2)° de- 
scribed a validation design for projective 
techniques which is quite comparable 
to that employed by Harrison, Cron- 
bach’s conclusions, which might also be 
said to apply to Harrison’s method, were 
that his type of approach (a) yielded a 
statistically sound test of significance, (b) 
“identifies objectively the accurate and 
inaccurate aspects of the prediction,” and 
(c) permitted “identification of the types 
of cases for whom prediction is relatively 

*Since Cronbach's article was published after 
the present study was completed, the particular 


design which he introduced was not originally 
considered in this investigation, 
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accurate” (2, p. 373). 

It should be noted that in this pro- 
cedure, the proof of validity hinges on 
the premise that the judges accept the 
statements of the interpretation as being 
similar to the items in the case analysis. 
The criterion for being “similar” is not 
stated in either of these articles, and from 
this procedure, no conclusion can be 
drawn as to the degree of similarity be- 
tween the two sets of items. This degree 
of similarity might be measured by a 
rating scale of agreement, as used by 
Krugman (11). The ultimate step in this 
validation procedure would be to demon- 
strate that the personality of the indi- 
vidual as inferred from the test and from 
a life situation could be described by 
the same statements, e.g., on a rating 
scale or on a check list of commonly 
used statements. 

The feature of this “item analysis” 
method of greatest import to our discus- 
sion is that on the surface it contradicts 
the assumption by Frank (3) and Vernon 
(18, 19), namely, that, since these descrip- 
tions deal with an integrated personality 
structure, they could not be validated 
piece by piece. This seeming contradic- 
tion might be explained, however, by the 
hypothesis that the validity of these sepa- 
rate items depends on the validity of the 
whole description. In the strictest sense, 
this hypothesis would mean that sepa- 
rate statements within the interpretative 
report would be valid only if the whole 
interpretation were valid. At least, it 
would indicate that if the interpretation 
as a whole is accurate, then the isolated 
statements drawn from the interpretative 
reports would be more likely to be accu- 
rate. It should be emphasized that this 
hypothesis refers to the validation of in- 
terpretations which stress the relation- 
ships between the various functionings 
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within the personality as a whole. 

However, the studies of Harrison (4) 
and by Cronbach (2) do not supply any 
direct evidence in support of this hy- 
pothesis, since neither study provided a 
test of the accuracy of the interpretations 
as integrated, whole reports. Harrison 
did not state whether or not certain of 
his cases had significantly lower “va- 
lidity indices.”” Although Cronbach con. 
cluded that his procedure allowed “the 
identification of the types of cases for 
whom prediction is relatively accurate,” 
he did not describe these cases in the part 
of his study reported to date. It is pos- 
sible that in both of these studies, some 
interpretations were inaccurate in the 
description of the personality as a whole, 
and that, therefore, the items drawn from 
these interpretations failed to attain sig- 
nificance. 

Unfortunately, it cannot be deter- 
mined from either Harrison’s or Cron- 
bach’s articles exactly on what basis the 
interpretations were broken down into 
isolated items. One main criticism of 
these two studies is that no rationale is 
presented as a basis for the selection of 
the items. In fact, Harrison did not adopt 
a dynamic, structural approach to per- 
sonality in his interpretations, but stated 
that his approach to personality was 
“eclectic and emphasized common sense 
psychology” in contrast to Murray’s (13) 
theories or to psychoanalysis. Nor did 
Cronbach describe the theoretical bias ol 
his Rorschach interpreters, although he 
did indicate that a more complete report 
would follow his introductory article. 
Exactly what types of statements about 
personality were validated, or might be 
validated, in this fashion remains unde- 
termined. ‘There is no assurance that this 
methodological design is applicable to 
the validation of interpretations base 
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on a dynamic theory of personality. 


FE. PURPOSES AND PROCEDURES OF THE 
PRESENT STUDY 


The purposes of the present study 
were: 

1. To test the hypothesis (discussed 
above) that the validity of separate state- 
ments about personality, inferred from 
projective techniques, depends on the 


5 


accuracy of the interpretation as a whole. 

2. To determine whether a test of the 
validity of isolated statements is appli- 
cable to interpretations based on a dy- 
namic, structural concept of personality. 

3. To determine whether the person- 
ality of the individual as inferred from 
the test protocol and from the criterion 
situation could be described by the same 
set of statements. 
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A. THE SELECTION OF THE RORSCHACH 
‘TECHNIQUE 


N ORDER to study the applicability of 
I two methods of validation, it was 
essential that these approaches be tested 
on a projective technique of relatively 
accepted validity. After due considera- 
tion, the Rorschach technique was 
chosen, mainly on the strength of a 
comparatively longer and more varied 
background of validation studies. For a 
complete review of these investigations 
of the validity of the Rorschach, the 
reader is referred to the three articles 
by Hertz (5, 6, 7). 

Predictive studies of the accuracy of 
Rorschach interpretation have not been 
as numerous, nor have the results been 
as uniformly positive, as those reported 
for the other types of approaches. Other 
than the studies using the matching 
method—which is under consideration 
here—most statistical studies have at- 
tempted to correlate isolated Rorschach 
signs with manifest. behavior, usually 
with negative results. In regard to these 
studies, Hertz remarked, “The abortive 
dissection of the psychogram in the 
search for static factors in isolation has 
distorted the [Rorschach] method” (6, 
P- 549): 

In addition to summarizing the vari- 
ous studies containing evidence of the 
validity of the Rorschach, Hertz pre- 
sented many positive criticisms of these 
studies and recommendations for further 
investigation. In particular, she stressed 
the need for more experimental and dif- 
ferential studies to evaluate the various 
hypotheses underlying the interpreta- 
tion of the Rorschach. The main purpose 


CHAPTER II 


THE TEST, THE CRITERION, AND THE SAMPLE 


of her review was to stimulate sound 
experimental design in these studies. 


1. The Administration of the Test 


The method of administration fol- 
lowed the procedure described by Klop- 
fer and Kelley (10). All the examiners 
took particular pains to secure a thor- 
ough “inquiry” into the features of the 
blots which elicited the subjects’ re- 
sponses. Probing and suggestive ques- 
tions were avoided, however, until the 
“testing of the limits.” Twenty-one of the 
subjects were tested by the author; the 
other seven subjects had been previously 
tested by other experienced administra- 
tors. 

2. The Protocols 


The validity of a Rorschach interpre- 
tation depends both on the quantity and 
quality of the subject’s responses. Most 
of these records offer a wealth of varie- 
gated responses as raw data for the in- 
terpreter. The completeness of the rec- 
ords was, to some extent, assured by the 
technique of administration, i.e., by the 
thorough inquiry. The quality of the 
protocols may also have been affected by 
the nature of the sample; pessibly the 
patients had been selected for psycho- 
therapy because they were comparatively 
more responsive and less restricted in 
their functioning. 


B. THe SusBjects 


In three Veterans Administration in- 
stallations, the Rorschach was adminis- 
tered to all patients who were currently 
receiving psychotherapy and to whom 
the Rorschach had not previously been 
given. Of these 28 subjects, 11 were in a 
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neuropsychiatric hospital, 11 were from 
an outpatient clinic, and 6 were attend- 
ing a nearby university clinic. These sub- 
jects ranged in age from 19 to 42 years, 
with a mean age of 28 years. Ten of the 
patients were Classified as psychotic, 16 
as neurotic, and 2 had other diagnoses. 
Except for the differences in diagnostic 
classification (see further discussion be- 
low), the character of this sample did 
not appear to have any direct bearing 
on the study of these two methods of 
validation. 


C. THe CRITERION 


Obviously, the validation of a projec- 
tive technique depends on the use of a 
criterion description which is comparable 
in nature to the test interpretation, and 
which is based on an adequate study of 
the individual. While the functioning 
of the individual’s personality may be 
inferred from his manifest behavior as 
summarized in a life history or factual 
case study, this functioning may be ob- 
served even more directly and intimately 
in a psychotherapeutic study, i.e., in the 
individual’s expression of his feelings and 
attitudes during psychotherapy, and in 
his emotional reactions to the psycho- 
therapeutic situation. This criterion has 
been used in at least two major clinical 
validations of projective techniques: 
Hertz and Rubenstein (8), and Tomkins 
(17). It was also recommended by Rosen- 
zweig (15) in his outline for a compre- 
hensive study of the validity of the Ror- 
schach. 

In the present study, the Rorschach 
interpretations were compared, in the 
two validation methods, with the thera- 
pists’ impressions of their patients. The 
seventeen therapists acted as the judges 
in both validation experiments, i.e., they 
selected the Rorschach reports which 
matched their patients in the matching 


experiment and described their patients 
in terms of the choices on the item check 
list. Thus, the validation judges were 
able to evaluate the Rorschach interpre- 
tations on the basis of an intimate and 
extensive knowledge of the subject, in- 
stead of having to reply on the basis of 
a summarized sketch compiled by a dis- 
interested party. 

The therapy which the patients were 
receiving was psychoanalytic in nature, 
i.€., its purpose was to reveal to the pa- 
tient his unconscious attitudes and moti- 
vations through an analysis of his emo- 
tional reaction to the therapeutic situa- 
tion itself. The purpose of this thera- 
peutic study of the patient’s underlying 
attitudes and motivations may be con- 
sidered equivalent to the aim of Ror- 
schach interpretation. In fact, these ther- 
apists frequently requested a Rorschach 
report on their patients’ personalities as 
an aid in planning treatment (excepting 
the patients who were subjects of this 
experiment). A majority of the thera- 
pists were also expert in the administra- 
tion and interpretation of the Rorschach 
technique. 

In the main, the therapists’ impres- 
sions of their patients were derived from 
frequent contact with them. The total 
number of therapeutic interviews at the 
time the therapist made his judgment 
ranged from 6 to go, with a median of 
19 interviews: only 4 subjects were inter- 
viewed less than 10 times, while 8 had 
been interviewed over go times. In 20 of 
the 28. cases, the Rorschach was adminis- 
tered when therapy was in a beginning 
stage, 1.e., before the fifth interview: the 
median number of interviews at the time 
of testing was 1, with a range of o to 4o. 
A median of 15 weeks elapsed between 
the time of testing and the time of judg- 
ment; in no case was there less than a 
7-week interval, and in 2 cases, the in- 
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terval was over go weeks. During this 
period, the therapists interviewed their 
patients between 5 and 80 times, with a 
median of 11 interviews occurring be- 
tween testing and judging. As to the fre- 
quency of these interviews, 6 of the cases 
were seen 3 or more times weekly, 5 
others were seen twice weekly, and all 
but 2 patients were seen regularly at 
least once a week. These 2 cases were 
interviewed frequently, but at irregular 
intervals. The minimum opportunity 
which a therapist had to observe his 


PALMER 


patient was 6 interviews, occurring at 
irregular intervals, over a period of 16 
weeks. The maximum observation oc- 
curred in a case where the patient was 
interviewed go times, 3 times a week, 
over a 30-week period. In summaty, it 
may be said that the therapists’ impres- 
sions were derived after extensive and 
frequent contacts with the subjects and 
may be considered comparable, in their 
theoretical approach to personality, to 
the Rorschach interpretations, 
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CHAPTER III 
THE MATCHING APPROACH 


N BRIEF, the matching experiment con- 
I sisted of the following procedures: 

1. Interpretative reports were prepared 
from each of the Rorschach records by 
one interpreter (the author). 

2. The reliability of these reports was 
checked in two ways: by having the re- 
ports matched to the protocols, and by 
having them matched to a duplicate set 
of reports which had been prepared by 
another psychologist. 

3. In order to test the reliability of the 
therapists in their matching technique, 
they were given a group of five sample 
interpretations from which they had to 
select the one which matched a corre- 
sponding sample case analysis. 

4. For each patient, a matching group 
was chosen, consisting of the interpreta- 
tion of that patient’s Rorschach, referred 
to hereafter as the experimental interpre- 
tation; and of four other interpretative 
reports, to be referred to as alternates. 

5. Each therapist: was then asked to 
select, from the group of five reports, the 
one report which he believed most closely 
represented his patient; subsequently, 
the therapist made a second choice 
among the remaining four reports. 


A. THE NATURE OF THE INTERPRETATIONS 


In order to standardize the method of 
interpretation and the style of the inter- 
pretative reports, all the protocols were 
interpreted by one person, the author. 
These interpretations were based solely 
on an individual's responses to the test 
material and on his behavior during the 
administration of the test.1 The method 


"The patient’s behavior during the testing (as 
distinguished from his responses to the materials) 


of interpretation followed that outlined 
in Klopfer and Kelley (10), particularly 


in the scoring of the responses and in . 


the preliminary analysis of the psycho- 
gram. The conceptual framework em- 
ployed throughout this process of inter- 
pretation was, broadly speaking, psycho- 
analytical. As far as possible, these de- 
scriptions were couched in everyday 
idiom, and both Rorschach and _ psycho- 
analytic terms were avoided. 


B. THe RELIABILITY OF THE 
INTERPRETATIONS 


Since all of the interpretative reports 
employed in the matching experiment 
were prepared by one person, it was nec- 
essary to determine whether these inter- 
pretations were reliable in the same sense 
that the consistency and accuracy with 
which one scores a psychometric instru- 
ment might be checked. As a rough test 
of this reliability, three judges, skilled 
in Rorschach interpretation, attempted 
to match the reports to the protocols, in 
groups of five each. Since this study was 
directed at the reliability or consistency 
of fully verbalized interpretations rather 
than of standardized symbols or scores, 
no attempt was made to check the au- 
thor’s scoring. These judges were instead 
presented with the unscored responses 
and asked to match these directly to the 
author’s statements about the various 
subjects’ personalities. All three judges 
were 100 per cent successful in this 
matching. 

Despite this positive result, it was pos- 


was not recorded. Since most of the records were 
interpreted some time after the test administra- 
tion, little if amy account was taken of this 
factor. 
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sible to question the reliability of these 
interpretations, i.e., whether they were 
similar to descriptions derived by some 
other interpreter. As a further check on 
this reliability, the protocols were inter- 
preted independently by another psy- 
chologist.2, Three judges matched, with 
complete success, the first five interpreta- 
tions of this second set of reports with 
the five corresponding interpretations by 
the author. Since this result coincided 
with the results of Krugman’s (11) more 
comprehensive study of Rorschach reli- 
ability, the success of this single match- 
ing was considered sufficient indication 
of the reliability of the interpretations 
used in the present study. 


C. THe RELIABILITY OF THE THERAPISTS 
IN MATCHING 


Prior to the matching of the Rorschach 
interpretations in the main part of this 
research, the therapists were briefly 
trained in the use of the matching tech- 
nique. In order to check the reliability 
of the therapists in matching, the author 
prepared a case analysis on a patient not 
included in the validation study. This 
patient’s Rorschach record was _ inter- 
preted independently by the interpreter 
who had participated in the study of the 
reliability of the interpretations. Using 
this case analysis as a criterion, ten of 
the therapists attempted to select this 
experimental interpretation from among 
four alternative interpretations (previ- 
ously prepared by this other interpreter). 

On this trial, six of the therapists 
matched the sample interpretation cor- 
rectly on their first choice, and three 

*The author wishes to acknowledge the pa- 
tient assistance of Mr. Mervin Freedman, Mr. 
Patrick Sullivan, and Mr. William Cook of The 
University of California who acted as the judges 


here, and of Mr. Allen Dittmann, who prepared 
the second set of reports. 
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others indicated it as their second choice, 
In this instance, successful matching on 
first choice could be expected by chance 
in two cases, i.e., two out of ten times. 
When first and second choices were con- 
sidered, chance matchings might occur in 
four instances in ten matchings. Accord- 
ing to the tables of “General Term of 
Poisson’s Exponential Expansion” in 
Pearson (14)* the obtained results of both 
the first choices alone (six correct match- 
ings) and of the two choices combined 
(nine correct matchings) are significant 
beyond the 5 per cent level of confi- 
dence, Admittedly, this limited study of 
the reliability of the therapists in match- 
ing a single case was not completely com- 
parable to the matching study of the 
validity of the twenty-eight cases, as de- 
scribed below. Still, in view of the posi- 
tive results of this brief reliability study, 
it seems reasonable to expect that these 
therapists would be approximately re- 
liable in other matching experiments— 
particularly one in which they would be 
more familiar with the criterion, i.e., 
their own patients. 


D. SELECTION OF THE MATCHING GROUPS 


As noted in tion I, the results of 
a matching study depends to a large ex- 
tent on the variability among the de- 
scriptive reports which constitute the 


* Throughout this study, many of the resultant 
proportions of chance agreement were so small 
that their distribution was thought to be con- 
siderably skewed and platykurtic. The use of a 
standard error of a proportion and its interpre- 
tation in terms of the normal probability inte- 
gral would have yielded erroneous probabilities. 
It was thought, however, that the computation of 
exact binomial probabilities would have required 
more effort than theix usefulness justified, so 
approximations to these probabilities were ob- 
tained from the Poisson distributions, This dis- 
tribution is useful in approximately binomial 

robabilities when p is small in comparison to q, 

ut where the possible number (of agreements, 
in our case) is finite. 


matching groups. In contrast to previous 
investigations which also made use of the 
matching method, the present research 
included an attempt to control the heter- 
ogeneity of the matching groups. This 
particular step in the present research 
may, therefore, bear some detailed ex- 
planation. 

The purpose of this step in the matching pro- 


cedure was to select matching groups having the 
same degree of heterogeneity. It seemed desirable 


that the alternate interpretations to be included 


in a group with the experimental interpretation 
should be neither very different from nor very 
similar to that experimental interpretation. To 
achieve this degree of heterogeneity, it was neces- 
sary to compare each interpretation with all 
other interpretations which might appear with- 
in a group as an alternate. In order to estimate 
the differences between interpretations, the fol- 
lowing rough rating scale was adopted: 

SS—Both interpretative reports describe the 
same basic personality features and indicate 
many similar specific characteristics. 

S—Both reports describe similar basic personal- 
ity features but may differ in specific character- 
istics. 

SO—Both reports describe some similar basic 
features and some similar specific characteristics, 
but also differ slightly in both respects. 

O—Both reports differ as to basic features but 
present some similar specific characteristics. 

OO—Both reports differ in all respects. 

X—The reports are not comparable. 

For the sake of convenience, the first fifteen 
reports collected from the sample were rated on 
this seale prior to the last thirteen reports. Three 
judges* rated these reports on the above scale; 
each judge made an independent rating first, 
followed by a final pooled rating by all three 
judges. These ratings were made on the over-all 
description of the personality rather than any 
specific cues. Thus, two interpretations discussing 
latent homosexual trends as important to the 
personality picture but differing in most other 
respects, i.¢., in basic personality structures, 
might be rated at least O, if not OO. 

In selecting the matching groups, reports 
which rated SO with the experimental report 
were given preference; a few reports compared 
as O or S$ were also used in some groups, but 
none of those extremely different or similar 
were included. Thus, the matching groups were 


*The author appreciates the assistance of Mr. 
es Leary and Mr. Walter Klopfer in this 
task. 
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of appropriate heterogeneity pith regard to the 

experimental report. 

E. THE SEQUENCE IN WHICH THE Two 
APPROACHES WERE USED 


Since both the matching and the check 
list judgments were made by the same 
judges (i.e., the therapists), the sequence 
in which the two techniques were tested 
carried a possible contamination: a ther- 
apist who matched his case before using 
the check list might be influenced by the 
selected report when the time came for 
him to make choices on the check list. 
Or, if he used the check list first, he 
might acquire a set for the matching of 
the report. Although such bias could not 
be wholly prevented, its possible effect 
was taken into account by systematically 
varying the order in which the therapist 
performed the two tasks. Therefore, in 
fourteen of the cases (seven in each half 
of the sample), the therapist selected the 
report before he made choices on the 
check list—the sequence being reversed 
in the other fourteen cases. The possi- 
bility of this type of bias was further les- 
sened by the fact that the therapists 
always made the two judgments separate- 
ly, with an intervening period of two to 
three weeks. 


F. How THE THERAPIST MApE His 
SELECTIONS 


Each therapist was presented with one 
matching group for each of his patients. 
Each group consisted of five reports (one 
of which was derived from his own pa- 
tient’s Rorschach). The therapist was in- 
structed to select the one report which 
matched his patient. After this first choice 
was made, the therapist was asked to 
name a second choice from the remain- 
ing four reports. The selection of a sec- 
ond choice was requested in order to 
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allow for partial errors in matching, par- 
ticularly in those instances when a judge 
might be undecided as to which of two 
similar reports to select. 

Another factor which had to be con- 
sidered in this procedure was the possi- 
bility that the patient’s personality had 
been altered by the therapy which inter- 
vened between the time the Rorschach 
was administered and the time the thera- 
pist made his selection. Therefore, as he 
made his selection, the therapist was re- 
minded of the date of the test administra- 
tion, and he was asked to consider the 
patient’s personality as it had been at 
that previous time. 


G. RESULTS OF THE MATCHING 


In eleven of the twenty-eight cases, the 
therapists correctly selected the interpre- 
tation of their patient’s Rorschach as a 
first choice from the matching groups; 
only two more were correctly selected as 
second choices. In terms of chance ex- 
pectancy (using Poisson’s tables) this re- 
sult is significant beyond the 3 per cent 
level of confidence. Using Vernon’s (19) 
formula for the coefficient of contingency, 
C is equal to .434 + .078. 


Although this matching was above chance, the 
relationship between the Rorschach and criterion, 
indicated by this C, was considerably lower than 
reported in previous studies. Vernon (18) found 
a C of 833+ .047.5 Krugman (11) reported a C 
of .850. Both studies differ from the present in- 
vestigation in two important aspects: 

1. They did not specifically control the hetero- 
geneity of their matching groups. It seems rea- 
sonable to presume that in the present study 
the control of this factor created a more difficult 
task for the judges, and consequently, a more 
acute test of the Rorschach reports. 

2. The previous studies used equal numbers of 
reports and case analyses, while the present 
investigation employed a five-to-one matching. 
Thus, in the present study, the judges were 
forced to differentiate among five reports, with 


*Vernon reported a PE of .0314, converted 


here for purposes of comparison to a standard 
error. 
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only one criterion as a basis of judgment; in 
this sense, the chance of success was probably 
much smaller than in the previous studies. 

Considering the factor of a more differentiating 
task for the judges, who therefore had less 
chance for successful matching, the degree of 
validity obtained in the present study may per- 
haps be regarded as comparatively more signifi- 
cant than the findings in the previous studies 
which did not include these controls. 

The results of the matching experiment might 
also have been affected by other variables in 
the nature of the sample or in any of the proced- 
ures used in collecting and presenting the data. 
Seven variables were considered as possibly af- 
fecting the results of this matching, namely: (a) 
the type of installation (hospital or outpatient); 
(6) the psychiatric diagnosis (psychotic or “other 
neuropsychiatric disorder”); (c) the total numbers 
of interviews (above or below the median of 19); 
(d) the numbers of interviews after testing 
(median or 11 interviews); (¢) the frequency of 
the interviews (weekly or more frequent); (f) the 
order in which the Rorschachs were administered 
(a difference in results was indicated between the 
two halves of the sample); and (g) the judgment 
which the therapist made first (matching or check 
list). A study of the effect of these seven variables 
was made to discover if they had any possible re- 
lationship to the matching results (see Table 1). 

Only one of these differences, the order in 
which the Rorschach tests were administered 
—and interpreted—is significant at less than the 
1 per cent level of confidence. A possible ex- 
planation of this difference is that the reports 
in the last half of the sample might have been 
more incisive descriptions than the first fifteen 
interpretations. The ratings of heterogeneity, 
which were made in the procedure for selecting 
the matching groups, provided some measure of 
the qualitative differences among the reports— 
at least within each of the two halves of the 
sample, but, unfortunately, not over the entire 
sample. The results of this rating procedure, as 
shown in Table 2, indicate that in both halves of 
the sample, a significantly greater number of the 
comparisons were rated as different from one an- 
other (O or OO) than might be expected if the 
distribution of ratings had been even; on the 
other hand, the percentage of § and SS ratings 
was much less than expected. Thus, the efforts of 
the interpreter to achieve distinctive reports 
were sustained within each half of the samples. 
Whether or not this distinctiveness increase 
progressively from one half of the sample through 
the next cannot be stated conclusively inasmuch 
as not each interpretation was paired with every 
other interpretation throughout the whole sam- 
ple. However, in view of the fact that no signili- 
cant difference existed between the two halves 
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TABLE 1 


DIFFERENCES, IN THE PROPORTION OF CASES MATCHED CORRECTLY, BETWEEN VARIOUS 
CHARACTERISTICS OF THE SAMPLE, AND BETWEEN VARIOUS PROCEDURES IN MATCHING 


Cases Matched Correctly 


Groups Compared 
N No. 
Hospital patients II 4 
Outpatients : 17 7 
Psychotic patients 10 2 
Other NP patients 18 9 


Total Interviews: 
Over 19 14 5 
Under 19 14 6 


Interviews after testing: 


Over 11 14 6 

Under 11 14 5 
Frequency of interviews: 

Once weekly 15 6 

Over once weekly 13 5 


Cases I-15 15 
Cases 16-28 13 


Check list first 14 
Matching first 14. 


un on 


Prop. Diff. CR P 
.36 .05+.27 <r 
-41 
.20 -30.14 2.15 .03 
.50 
-36 .0o7+.18 
-43 
+43 .0o7+.18 <I 
-36 
.40 .38 <1 
.38 
.56+.16 <.or 
-69 
-14+.19 <1 
.36 


of the sample in the proportion of O + OO rat- 
ings, it may be considered that the cases of the 
second half were no more distinctive, as com- 
pared among themselves, than those of the first 
half. 

The second of these variables which showed 
a significant difference in the matching results 
was the diagnostic classification of the patients. 
Fewer of the cases diagnosed “psychotic” were 
matched correctly than those classed as “neuro- 
tic’ or in other neuropsychiatric categories. Al- 
though the number of patients in these classifi- 


TABLE 2 


DIFFERENCES BETWEEN OBTAINED AND EXPECTED RATINGS or S+SS anp 0+00 


(Assuming an expected chance distribution of equal 
proportions of SS+ S, SO+X and 0O0+0.) 


cations was too small for further computation 
of differences, it was noted that seven of the ten 
psychotic cases fell in the first half of the 
sample. These results, if meaningful, would 
indicate that the interpretations of the Rorschach 
of the psychotic patients may have been less 
differentiating ones. In view of the fact that 
psychotic patients (other than paranoid types) 
often give vague and diffuse responses, it is to be 
expected that these interpretations would be 
less meaningful and distinctive. Such responses 
from the psychotic patients are consonant with 


Compari- 
sons 


Rating | Obtained | op tained | Expected 


7% Diff. CR P 


S+SS 16 
0+00 44 


I-15 105 


S+SS 9 


16-28 78 
0+00 38 


15.2 —17.8 3-73 <.o1 
33+4.47 

48.8 +10.8 2.06 <.05 

4.39 <.O1 
33+4-9 

48.7 +15.7 <.01 
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the theory of personality used here, i.e. that 
inadequate perceptual differentiation is equated 
with psychoses, However, this concept is not 
helpful in distinguishing one psychotic patient 


from another, as was required of the matching 
judges. If the judges operated on the basis of 
this concept also, then the criteria may have been 
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as nondifferentiating as the Rorschach. Furthe 
study of the psychotic individual may be 
quired, by both the Rorschach and other methog; 
of observation, before a higher validity of inte, 
pretation can be demonstrated by the matching 
method. 
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HE CHECK LIST approach consisted of 
T the following steps: 

1. A list of thirty-four multiple choice 
items was constructed, consisting of state- 
ments commonly used in interpretative 
reports and in psychotherapeutic analy- 
SIS. 

2. The reliability of the therapists in 
the use of this check list was determined, 
using a sample case analysis as criterion. 

3. Four Rorschach interpreters inde- 
pendently checked their choices on the 
multiple choice items for the twenty- 
eight Rorschach protocols. The relia- 
bility of these Rorschach judges was 
determined by computing the signifi- 
cance of the number of agreements be- 
tween these judges, for each item. 

4. The therapists checked choices on 
each item on the basis of their impres- 
sion of their patients, The validity of the 
Rorschach judges’ choices was then de- 
termined by computing the significance 
of the number of times that they agreed 
with the therapists on each item. 


A. SELECTION OF THE ITEMS 


In order to obtain a list of multiple 
choice items representative of the many 
abstractions used in Rorschach interpre- 
tation, each of the major categories of 
personality utilized in the interpretative 
reports was represented by at least one 
item. These six major areas of personality 
were as follows: (a) inner drives and 
attitudes, (b) emotional reactions and 
relationships, (c) sensitivity to emo- 
tional stimuli, (d) intellectual function- 
ing and reality testing, (e) sexual attitudes 
and identification, and (f) anxiety and 
defenses against anxiety.’ Each of these 
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major categories or areas was further 
considered in four subdivisions or “di- 
mensions”: (a) the frequency or extent 
to which these areas were represented in 
his reaction to the test materials or to 
psychotherapy; ()) the characteristic type 
or nature of each area; (c) the role which 
each area played in the total pattern of 
the personality; (d) the control or manner 
in which the individual handled the at- 
titude or reaction in question. 

The questions asked in the interpreta- 
tive reports concerning the individual’s 
attitudes toward his identity and his 
inner motivations were represented on 
the check list by those items referring to 
fantasy life and inner drives, as follows: 


(Quantity) No. 17. “Expression by the indi- 
vidual of his inner needs and drives, i.e., his 
striving for satisfaction of these drives, is: almost 
completely absent,” to, “directly impulsive, show- 
ing an infantile lack of control.” 

(Role) No. 23. “Such inner fantasy life as the 
individual may allow himself is utilized for, or 
functions in his personality structure as: A. An 
internalization of certain unacceptable feelings, 
not permitted in overt behavior, e.g., for intro- 
jection of hostility in an intrapunitive manner. 
B. An attempt to organize and handle outer be- 
havior in an integrated manner. C. A retreat 
from nearly all environmental frustrations, es- 
pecially those in interpersonal relationships, with 
a handling of such relationships on a fantasy 
level. D. Very little, being poorly developed. E. 
Very little, being a source of anxiety in itself.” 

(Control) No. 13. “The method by which the 
individual handles and controls his inner emo- 
tional drives is chiefly: A. By fantasy solutions— 
possibly by divorcing such feelings from reality. 
B. By creative use of his energies, in a sublimated 
manner. C. By repressing them in a rigid and 
constricted manner. D. By direct release in overt 
behavior. E. By attempting to intellectualize, 
depersonalize or otherwise detach them from 
him emotionally.” 


Closely related to this general area of 
inner motivation are the individual’s 
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attitudes toward his sexual functioning, 
which were sampled on the check list 
by the following items: 


(Quantity) No. 6. “The extent to which the 
individual enters into heterosexual relationships: 
is almost completely nil,” to, “is so exaggerated 
as to pervade much of the individual's behavior.” 

(Type) No. go. “The following attitude may 
be considered as the ‘basic’ one with which the 
individual regards his own sexual or ‘sexualized’ 
behavior: A. As an aggressive (sadistic act. B. 
As a dangerous (castrating) act. C. As a passive, 
receptive (incorporative) act. D. As a demonstra- 
tion of potency, an egotistic self-assertion, (auto- 
erotic or exhibitionistic). E. As normal and ac- 
ceptable (genital supremacy) .” (A—D assume in- 
fantile sexual fixations or conflicts.) 

No. 32. “The individual’s general identification 
in most sexual and social roles is: A. with a 
dominant male figure. B. With a dominant fe- 
male figure. C. With a passive male figure. D. 
With a passive female figure. E. Without a 
definite character and/or extremely ambivalent.” 

(Role) No. 10. “Homosexual relationships are 
utilized by the individual for, or play a role in 
his personality as: A. An integrated and mature 
part of his social behavior. B. A denial of rejec- 
tion by other males. C. A denial of rejection by 
females. D. A minor role (e.g., for further satis- 
faction of narcissistic needs). E. An assertion of 
identification as to sexual role.” 

(Control) No. 24. “The chief method by which 
the individual handles his homosexual relation- 
ships is: A. By fairly overt emotional attach- 
ments possibly including sexual satisfactions. B. 
By repression of such feelings and/or avoidance 
of such relationships. C. By sublimating such feel- 
ings into socially acceptable channels of behavior. 
D. By retreating into fantasy solutions. E. By 
intellectually detaching the emotional aspects, 

depersonalization.” 

No. 34. “The individual handles, or reacts to, 
possible heterosexual relationships (or needs for 
such relationships) chiefly by: A. A retreat into 
fantasy (without necessarily breaking with re- 
ality). B. By accepting social restrictions, and 
sublimating where necessary. C. By repressing 
such drives, and/or depersonalizing them, de- 
taching the emotional aspects of such relation- 
ships. D. By affective outburst—such as overt 
anxiety and panic, etc. E. By breaking with 


reality.” 

The manner in which the individual 
perceives and accepts the pressure of his 
environment was represented in the fol- 
lowing items, which dealt with inter- 
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personal relationships and emotional 
reactions: 


(Quantity) No, 27. “The degree to which the 
individual allows himself to become involved jn 
emotional relationships with others is: a very 
limited involvement of any type,” to, “a purely 
volatile and explosive reaction.” 

(Type) No. 25. “The emotional tone or affect 
which the individual displays in his emotional 
attachments with others is most often: A. Warm 
and spontaneous. B. (Absent.) C. Cold and 
detached. D. Hostile and oppositional. E. Forced 
and artificial. ? 

(Role) No. 33. “Involvement in active emo. 
tional, interpersonal relationships serves the indi. 
vidual as, or plays a role in his personality 
structure as: A. A method of compensating for 
the inadequacies felt within himself. B. A release 
mechanism for the satisfaction of inner drives. C. 
A minor role—in an intraversive adjustment, 
under accumulated frustration or increasing 
environmental stimulation only. D, (None.) E. 
As a mature and integrated part of his behavior.” 

(Control) No. 19. “The principal method by 
which the individual handles and controls his 
emotional reactions in his interpersonal relation- 
ships is: A. By integrating them in a mature 
manner with other personal needs. B. By rigidly 
avoiding and denying the emotional aspects 
(isolation of emotion). C. By ignoring the reality 
of such an emotion and autistically withdrawing 
into fantasy solutions. D. By immature, and 
possibly aggressive, reactions. E. By depersonal- 
izing such situations through intellectualizing or 
rationalizing.” 


Four other items were constructed re- 
quiring judgment relative to the indi- 
vidual’s sensitivity to environmental stim- 
ulation: 


(Quantity) No. 8, “The extent to which the 
individual allows himself to be receptive to the 
affective feelings of others, or to other emotional 
stimulation: A. Is limited and tenuous, chiefly 
when socially approved. B. Is practically absent. 
C. Is such that the individual is acutely aware 
of the emotional aspects of a situation. D. Shows 
a well-balanced and integrated sensitivity and 
tact. E. Shows a tendency to be unduly sensitive.” 

(Type) No. 9. “The individual’s most charac- 
teristic reaction to the affective feelings of others 
and to the emotional stimulation from his en- 
vironment, is: to be indifferent and disinterested,” 
to, “to be overtly sensuous.” 

(Role) No. 16. “Sensitivity to the emotions of 
others or to other emotional stimuli is utilized 
by the individual for, or plays a role in his 
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personality as: A. (None.) B. A counter-reaction 
to repressed hostility. C. A withdrawal from 
frustration, from a more active emotional in- 
volvement. D. An integrated part of a mature 
handling of social relationships. E. A primary 
source of guilt and anxiety.” 

(Control) No. 2. “The way in which the indi- 
vidual controls and handles possible sensitivity to 
emotional stimulation, especially to the feelings 
of others, is usually: A. By rigid repression and 
restriction of such sensitivity. B. By attributing 
such stimulation to his own inner needs (by 
introjection). C. By reactively denying such feel- 
ings in aggressive, hostile behavior. D. By divorc- 
ing such feelings from reality and/or by fantasy 
solutions. E. By acceptance and integration of 
such sensitivity in social relationships.” 


From the area of intellectual function- 
ing and of reality testing, the following 
items were derived: 


(Quantity) No. 1, “The wealth of the individ- 
ual’s intellectual activity may be characterized as 
generally: impoverished, and tending to be per- 
serverative,” to, “having a wide range of inter- 
ests, often being rich and original in content.” 
No. 7. “The individual’s intellectual productivity 
may be estimated as generally: very limited,” to, 
“extensive.” 

(Type) No. 15. “The individual's intellectual 
‘approach to a problem or situation: A. Usually 
shows a tendency to abstract and over-generalize, 
without sufficient attention to everyday affairs. 
B. Tends to be overly critical, analytical, possibly 
picayunish. C. Usually. shows a fair ability to 
conceptualize, but with adequate attention to 
practical concrete matters. D. May contain some 
evidence of delusional thought processes, forcing 
relationships between facts or distorting reality. 
E. Is most often a matter-of-fact approach, tend- 
ing to be overly concrete.” 

No. 18. “The individual's ties to reality may be 
classified as chiefly: very strong, as never per- 
mitting any vagueness” through, “adequate—but 
not overly concerned with reality testing,” 
through, “so tenuous as to easily become inade- 
quate,” to, “quite inadequate and/or absent.” 

(Role) No. 22. “Intellectual functioning is 
utilized by the individual for, or has a principal 
function in his personality structure as: A. A 
rigid defense against the release of inner drives 
and/or emotional ties with others, by depriving 
them of their emotional tone. B, A mature and 
normal mode of controlling himself and his en- 
vironment, C, As an aid to autistic thinking, e.g., 
in delusional types of solutions. D. A highly 
aggressive, critical defense mechanism. E. Only a 
minor role, e.g., as an aid to immediate satisfac- 
tion of narcissistic needs.” . 


(Control) No. 5. “The individual's contact with 
reality appears weakest: A. In his creative, inner 
fantasy life. B. In his active, and potentially 
aggressive, interpersonal relationships. C. In his 
sensitivity to emotional stimulation. D. In seem- 
ingly impersonal situations (to which affect has 
been displaced). E. In his release of instinctual 
drives.” 


The items included on the check list 
under the rubric of anxiety correspond 
in part to those considerations given ego- 
functioning above: 


(Quantity) No. 28. “The degree to which the 
individual shows feelings of generalized disturb- 
ance may be estimated as: seldom more than a 
minimal and occasional uneasiness,” to, “states 
of overwhelming panic.” 

(Type) No. 3. “The individual's expression of 
feelings of generalized disturbance may be char- 
acterized as: A. A free-floating type of anxiety 
state. B, A feeling of inner tension and conflict- 
guilt feelings. C. Overt depression. D. A sense of 
frustration and disappointment. E. (Relatively 
absent.)” 

(Role) No. 21. “The effect of anxiety and/or 
guilt feelings on the individual's personality 
structure constitutes: no noticeable effect,” to, “a 
gross breakdown of most functioning.” 


The different defenses were considered 
more specifically in the following items: 


(Projection) No. 4. “Projection of guilt feelings 
onto others or the environment is used by the 
individual as a method of averting anxiety to 
the following degrees: very rarely,” to, “exten- 
sively.” 

(Rationalization) No. 11. “The individual uses 
rationalization and justification as an intellectual 
evasion of anxiety to the following degree: very 
rarely,” to, “extensively.” 

(Obsession) No. 12. “The individual uses com- 
pulsive behavior or obsessive thinking as a magi- 
cal and ritualistic denial of anxiety to the follow- 
ing degree: very rarely,” to, “extensively.” 

(Displacement) No. 14. “The _ individual 
attempts to avert anxiety by displacement of 
emotional content to some more ‘neutral’ situa- 
tion: very rarely,” to, “extensively.” 

(Withdrawal) No. go. “The individual attempts 
to avert anxiety by fantasy solutions and/or by 
withdrawal from contact with reality to the 
following degree: very rarely,” to “extensively.” 

(Normal reaction) No. 26. “The individual 
uses anxiety as a normal ‘warning signal’ of pos- 
sible frustration: very rarely,” to, “extensively.” 

(Acting out) No. 29. “The individual attempts 
to avert anxiety by ‘acting out’ of frustrations 
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onto the environment, by negativism and aggres- 
sion, etc.: very rarely,” to, “extensively.” 

(Isolation) No. 31. “The individual attempts to 
avert anxiety by rigid isolation of all emotional 
aspects of a situation: very rarely,” to, “exten- 
sively.” 

B. ARRANGEMENT OF THE ITEMS 
ON THE List 


It is possible that, if the items had been 
presented in the logical order of the 
scheme shown above, a judge’s choice 
on one item might directly influence his 
choices on succeeding items, especially 
those within the same major category. In 
order to lessen the sequential effect, the 
items were presented in a random ar- 
rangement. 


C. THe NUMBER AND ORDER 
OF THE CHOICES 


For the sake of uniformity, five choices 
were listed for each statement. As was 
discovered afterward, this uniform num- 
ber of choices was an unnecessary restric- 
tion; in many instances, a larger number 
of choices would have offered the judges 
more opportunity to describe their pa- 
tients, and in a more accurate manner. In 
many instances, also, these five choices 
formed an obvious continuum, e.g., from 
“extensively” to “rarely,” or “well ad- 
justed” to “very disturbed,” etc. Since a 
judge’s choices on one item might well be 
influenced by the position on this con- 
tinuum of his choices on previous items, 
the order of choices was varied in a ran- 
dom manner from item to item. 


D. THE INSTRUCTIONS 


The judges were instructed to make 
two choices on each item for each pati- 
ent. Second choices were requested, be- 
cause (a) it was possible that a pair of 
judges might agree, given two choices 
each, even though they might not agree 
on a first choice; and (b) some judges felt 
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less forced in their judgments when al- 
lowed two choices. 


E. RELIABILITY OF THE THERAPISTS IN THE 
OF THE CHECK LIsT 


Reliability of the therapists in the use 
of the check list was determined in the 
same manner as was their reliability in 
the matching procedure. Using a sample 
case analysis (described above) as a cti- 
terion, ten of the therapists indicated 
choices on each item of the list. The 
therapists had previously had some train- 
ing in the use of this type of list, in a 
trial run of a preliminary form. A rough 
estimate of the reliability of these thera- 
pists was obtained by assuming the agree- 
ment to be satisfactory when 5 or more 
therapists indicated the same first choice 
on an item—as describing this sample 
case history. When any two judges indi- 
cated the same two choices on any item, 
either as first or second choice, this was 
also noted as an agreement. For these 
“pairs” of choices, satisfactory agreement 
on an item was assumed when four or 
more judges used the same pair. On 
twenty of the items, five or more judges 
employed the same pair of choices. Al- 
though this reliability study was admit- 
tedly limited, it seems reasonable to as- 
sume that similar results. might have 
been obtained if a more extensive study 
had been possible. Strictly speaking, the 
results of this step in the check list ap- 
proach can only be taken to indicate that 
the therapists were able to agree satis- 
factorily on a majority of the items, using 
one sample case as a basis of judgment. 


F. RELIABILITY OF THE RORSCHACH JUDGES 
IN THE UsE OF THE CHECK LIST 


The 28 Rorschach records of the 
sample were judged independently on the 
check list by four experienced Rorschach 
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interpreters. These judges‘ received a 
brief training in the use of this check 
list on a sample Rorschach protocol. The 
reliability of these Rorschach judges in 
the use of the check list was considered 
in terms of the number of agreements, 
i.e., the number of times they indicated 
the same choice on an item regarding the 
same individual. For each item, the 28 
first choices of each judge were compared 
with those of every other judge—two 
judges at a time. Thus, six sets of com- 
parisons were made for each item. The 
first choices of each pair of judges were 
tabulated on a five-square table, such 
that the agreements fell on the diagonal. 
The degree of agreement expected by 
chance was then computed on the as- 
sumption that the two sets of judgments 
were independent, subject to the restric- 
tion of the observed marginal totals. 
Since in most instances this chance de- 
gree of agreement was a relatively small 
proportion of the total number of cases— 
usually less than one-third—the signifi- 
cance of the difference between this num- 
ber of agreements. expected by chance 
and the number of agreements actually 
obtained was again read from the tables 
of Poisson’s distribution (14). Where 
the expected number of agreements was 
larger than one-third of the total number 
of cases, the significance of this difference 
was computed in terms of the standard 
error of a proportion. 

All in all, the four judges of the Ror- 
schach agreed significantly about one- 
third of the time. Of the 204 comparisons 
for first choices (the choices of six pairs 
of judges compared on 34 items), 78 re- 
sulted in agreement at the 10 per cent 


*Mr. Walter Klopfer, Mr. J. Neil Campion, Jr.. 
and Dr. Claire Thompson of the University of 
California were kind enough to devote many 
hours assisting the author in making these judg- 
ments, 
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level or beyond; 51 of these were signifi- 
cant at the 5 per cent or beyond; and 26 
at the 2 per cent level or better. For the 
two choices combined, 63 of the obtained 
agreements were significant at the 10 
per cent level or beyond; 43 of these at 
the 5 per cent level; and 31 at the 2 per 
cent level or better. If we consider the 
rather abstruse wording of some of these 
items and the limitations on the number 
of choices, this degree of agreement is 
considerable. The large percentage of 
lack of agreement is not surprising in 
view of the fact that the judges were at- 
tempting to make almost unqualified 
statements about personality from such 
a restricted sampling of behavior, i.e., re- 
sponses to ten ink blot pictures. The 
alternative or second choice did not ap- 
preciably increase the agreement between 
the judges. 


Further analysis of the agreement and lack of 
agreement indicates that a fair number of items 
may be called reliable. If three significant agree- 
ments, i.e., agre¢ment by three pairs of judges, be 
granted as indicating satisfactory reliability for 
an item, then about one-third of the 34 items 
may be called reliable—12 for first choices and 11 
for the combined choices. An appreciable part of 
the lack of agreement may have been attributa- 
ble to some particular pair of judges. However, it 
was found that all pairs of judges agreed to 
about the same degree. The degree of agreement 
within each area and within each dimension, as 
shown in Table 3, is summarized as a propor- 
tion of the total number of comparisons made 
in that area or dimension which showed signifi- 
cant agreement. The area in which the propor- 
tion of agreement was highest was intellectual 
functioning (.667). This particular area has been 
given close attention in the interpretative 
method of Klopfer and Kelley (10). Besides, it 
has also been thoroughly discussed in the 
current literature; therefore, the comparatively 
strong agreement shown here is not surprising. 
Two items which were included in this area, but 
which failed to have three G. more agreements 

(Nos. 5 and 18) should properly belong to reality 
testing in general, rather than to specific intellec- 
tual functioning. Undoubtedly, a better defini- 
tion of reality testing should have been reached 


by the judges. 
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TABLE 3 


DisTRIBUTION OF SIGNIFICANT AGREEMENTS BETWEEN RORSCHACH INTERPRETERS 


ACCORDING TO AREA AND DIMENSION OF PERSONALITY FUNCTIONING , 


Frequency 
No. of | Propor- 


Type 


Area Compar- a of No. of No. of No. of No. of 
isons gree- | Items | Agree- | Items Agree- | Items | Agree- | Items Agr 
ment 
ments ments . 
Inner 
drives 18 17 2 13 t 
Sexual 36 -277 6 ° 30 3 10 ° 34 r 
attitudes 32 I 24 2 
Emotional 
reactions 24 .417 27 4 25 2 33 I @ ae 
Sensitivity 24 .458 & 3 9 I 16 I 6 
Reality 36 .667 I 6 15 6 22 5 2 
testing 7 5 18 2 
Anxiety 
reactions 


Number of comparisons 42 


Proportion of agreement .524 


42 36 36 


-328 .250 .606 


area: the one dealing with the individual’s inner 
life, including his motivations and incorporated 
attitudes (611). The significance of this agree- 
ment lies in the fact that this area is not as easily 
interpreted as that of intellectual functioning. 
This result might be explained by the hypothe- 
sis that the Rorschach is designed to tap these 
inner drives more than it does other areas. How- 
ever, the degree of agreement about these inner 
drives does not appear to be much stronger than 
that about outer emotional reactions (417) or 
sensitivity (.458). (The actual number of agree- 
ments was too small to permit statistical com- 
parison.) 

The lower percentage of agreement regarding 
emotional reactions and sensitivity may be traced 
to the disagreement among the judges about the 
dimensions of type and role. These two dimen- 
sions in these areas were the most difficult to 
restrict to five choices, and it seemed that greater 
agreement might have occurred if more choices 
had been provided. 

Although the agreement on items concerning 
sexual attitudes (.277) might also have been im- 
proved by expanding the number of choices, the 
basic difficulty was more likely the confusion over 
the concepts employed in these items, particul- 
arly the concept “homosexual.” This term was 
intended to refer to any type of attitude toward 
the same sex, rather than just to overt sexual be- 
havior. Although the judges understood this 
broader connotation, they tended to limit their 


thinking about it to the latter, more common 
meaning. 


The judges agreed reasonably well on another 


The notably low proportion of agreement on 
the items pertaining to anxiety and defenses 
(.166) (see Table 4) is again explicable by the 
fact of the limited number of choices. To para- 
phrase: all anxiety is scarcely divisible into five 
parts! A more serious source of disagreement was 
found to be the poor definition of terms in this 
area, Even after the use of the check Ist had 
been reviewed among the judges, such terms as 
“displacement” were not consistently applied: 
note the significant disagreements on this item 
(No. 14) particularly. Several of the judges ex. 
pressed the opinion that the items referring 10 


defenses often overlapped in meaning or were 
otherwise confusing. 


TABLE 4 


DISTRIBUTION OF SIGNIFICANT AGREEMENTS 
BETWEEN RORSCHACH INTERPRETERS, 
ACCORDING TO Type OF DEFENSE 


Item No. of 
Defense No. agreements 
Projection 4 2 
Rationalization II ° 
Obsession is I 
Displacement 14 —2 
Withdrawal 20 ° 
Normal warning 26 2 
Acting out 29 I 
Isolation 31 2 


Total No. of comparisons 


oo 


Proportion of agreement 166 
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Viewing the results according to di- 
mension, the highest agreement occurred 
on those items concerning the indi- 
vidual’s handling or control of his func- 
tioning (.606). Next on the scale of 
agreement came the dimension of fre- 
quency (.524). These results indicate 
that Rorschach ‘interpreters can agree 
among themselves on the two following 
aspects: the way in which an individual 
controls his impulses, and the degree to 
which he uses any particular control or 
defense or expresses an attitude. 

The difference in the percentage of 
agreement between control (.606) and 
role (.250) assumes an importance when 
one considers that both dimensions deal 
with relationships between reactions. 
Thus, the judges were in much better 
agreement as to the control relationship 
between various areas of personality than 
they were concerning the relative im- 
portance and relationship of a particular 
reaction or attitude in the individual's 
overall functioning. This lack of agree- 
ment about the concept of role may be 
due partially to inadequate phrasing of 
the items or limitations in the number of 
choices. But the fact should also be 
pointed out that Rorschach interpreta- 
tion underscores this concept of control, 
and that, too often, little attention is 
given the importance of such control or 
defense in the “economics” of the per- 
sonality. 

Thus, if it be granted that Rorschach 
theory emphasizes the perceptual func- 
tioning of the ego, then the comparatively 
higher reliability on the items covering 
intellectual functioning and control was 
to be expected. Conversely, these results 
may be taken to show a lack of agreement 
on those aspects which are less clearly 
defined in Rorschach theory, namely, the 
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contentual factors, such as sexual atti- 
tudes and types of anxiety. 


G. RESULTS OF VALIDATION ON THE 
CHECK LIsT 


The choices on each item by each 
of the four Rorschach judges were tabu- 
lated with those of the therapists—one 
pair of judges at a time; four differences 
between the obtained frequencies of 
agreement and that expected by chance 
were derived for first choices, and the 
significance of these differences was noted 
in terms of the approximation to bi- 
nomial probabilities read from Poisson’s 
tables. 

Considering first choices alone, only 9 
of the 136 comparisons (4 comparisons on 
the 34 items) resulted in significant dif- 
ferences, at the 10 per cent level of con- 
fidence or beyond. No significant agree- 
ment appeared for the combined choices 
(first and second choices together), Only 
5 of the differences on first choices were 
“positive differences,” i.e., the obtained 
agreement was larger than that expected 
by chance, while 4 were “negative differ- 
ences,” i.e., the obtained agreement being 
smaller than chance. Since, in a distri- 
bution of 136 differences, 13.6 might be 
expected to show such significance by 
chance, this small number of significant 
agreements cannot be considered to be 
of any statistical importance. The dif- 
ferences did not seem to occur in any 
meaningful pattern: no more than 1 dif- 
ference occurred on any one item; al- 
most an equal number of such differences 
occurred for each pair of judges; and 
these differences did not appear to have 
any relation to the scheme of personality 
areas and dimensions. 

In general, these results support the 
hypothesis that the check list type of 
approach is not applicable to the vali- 
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dation of the interpretations of projective 
techniques when these techniques are 
interpreted by means of a dynamic con- 
cept of personality. Such statements as 
those used in this check list are appar- 
ently meaningless, except in the context 
of an integrated descriptive report. 

Undoubtedly, the amount of agree- 
ment between the Rorschach judges and 
the therapists was dependent on the de- 
gree to which these two sets of judges 
agreed separately among themselves. The 
reliability studies discussed above showed 
that when one set of judges used one 
kind of data, either Rorschach records or 
therapy, the agreement in that instance 
was satisfactory. In terms of the number 
of items which showed significant agree- 
ment, the Rorschach judges agreed 
among themselves gs per cent of the 
time, i.e., on 12 items, while the thera- 
pists agreed among themselves 58 per 
cent of the time, or on go items. In view 
of these reliabilities, there would seem 
to be some chance of obtaining higher 
validity. At the same time, one might 
inquire as to why these isolated state- 
ments were not applicable to the valida- 
tion of Rorschach interpretation when 
they appear to be applicable to the study 
of its reliability. 

Qualitative examination of the reli- 
ability of the Rorschach judges and of the 
therapists may serve to explain why there 
was agreement within each respective set 
of judges but little agreement between 
the two sets. In fact, the results of this 
reliability study may aid in the explora- 
tion of the relationships between the 
isolated statements and the whole re- 
ports. 

As has been noted in the above discus- 
sion, the Rorschach judges agreed among 
themselves on those statements referring 
to the concepts which are most clearly 
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defined in Rorschach theory, namely, 
in the area of intellectual functioning 
and on the dimension of control. A}- 
though several other basic concepts are 
commonly recognized in Rorschach inter- 
pretation, these two concepts may be 
considered as the “axis” from which the 
whole pattern of the personality is 
evolved. It must be acknowledged that 
when other concepts are introduced in 
an interpretation, they are not based 
entirely on the relationship between 
those two primary inferences. However, 
the analysis of clues in the Rorschach 
protocol associated with other concepts 
than intellectual functioning or control 
is strongly influenced by the conclusions 
about these two concepts. 

Although the therapists used the same 
general framework of concepts as adopted 
by the Rorschach interpreters, there was 
no direct evidence as to which particular 
concepts within this framework were 
central in each therapist’s thinking. 
Since the therapy dealt principally 
with emotional relationships and with 
the analysis of emotional reactions, one 
might expect that the concepts concerned 
with this area of the personality deter- 
mined the therapist’s orientation. Thus, 
in considering a patient’s personality as 
observed during therapy, the therapist 
might be inclined to give weight only to 
such intellectual functioning as was di- 
rectly related to the patient’s emotional 
life. 

Although the Rorschach interpreter 
and the therapist may have had different 
concepts and clues in mind as they ana- 
lyzed their respective observations of the 
patient’s reactions, they were both at- 
tempting to infer a total picture of the 
individual's functioning, i.e., his per- 
sonality structure. Each set of judges used 
their particular clues and concepts con- 
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sistently and reliably when considering 
their respective data. The two sets of 
judges did not agree significantly on the 
concepts which were not central to their 
separate considerations, especially when 
these concepts were isolated from the con- 
text of the whole structural pattern—as 


was required on the check list. On the 
other hand, when the total picture of the 
individual was taken into consideration, 
agreement between the therapists and 
the Rorschach interpreter was obtained, 
as was demonstrated in the matching 
experiment. 
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E RETURN to the hypothesis which 
W assumed that validity of (or agree- 
ments on) specific statements is more 
likely to be found when the whole de- 
scriptive report is validated. Bearing in 
mind that of the 28 whole reports in this 
study, only 11 were _ satisfactorily 
matched or validated, then, little or no 
validity could be expected throughout the 
check list for the entire sample. On the 
other hand, more agreement would be 
expected on the check list items for those 
11 cases on whom there was also agree- 
ment on the whole reports. In order to 
test this hypothesis, the sample was di- 
vided into two groups, based on the re- 
sults of the matching experiments, i.e., 
the 11 cases correctly matched, and the 
17 cases in which matching was unsuc- 
cessful. 

As one test of the hypothesis, the sig- 
nificance of the agreements between the 
Rorschach and criterion judges was cal- 
culated separately for these two groups 
on each item of the check list. This di- 
vision yielded no significant agreement 
for any one of the items for either group, 
whether considering the first choices 
separately or the combined choices. These 
results were not surprising, considering 
the results of the validation of these items 
for the sample as a whole. In addition, 
a positive agreement significantly above 
chance was contraindicated because of 
the small number of cases in each of the 
two groups. 

A preliminary analysis of the data indi- 
cated that the obtained agreements on 
the check list for the 11 matched cases 
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THE RELATIONSHIP BETWEEN THE TWO APPROACHES 


were consistently greater than those {or 
the other 17 cases, although not enough 
to be statistically significant. This possible 
difference was again tested by contrasting 
the obtained agreements and disagree. 
ments for both groups on fourfold chi- 
square tables. Since the number of agree- 
ments on first choices was too small to 
permit this comparison for any item, a 
chi-square test was made on the first and 
second choices combined. No significant 
results were obtained on the whole. Only 
11 of the 136 comparisons attained signifi- 
cance beyond the 10 per cent level of 
confidence. Five of these chi-squares indi- 
cated a difference in the expected direc- 
tions, i.e., the correctly matched group 
had more agreements, but 6 were in the 
opposite direction, i.e., the group not 
correctly matched possessed greater agree- 
ment on the check list item in question 
than did the cases correctly matched. 
There was no indication that these re- 
sults were associated with any particular 
pair of judges or with any definite area 
of content of the items. 

The absence of any significant degree 
of difference between these two groups 
of cases indicates that the agreement be- 
tween the Rorschach and therapy judges 
on the specific items had no relation to 
their agreement on the whole reports. 
The lack of significant agreement be- 
tween the Rorschach judges and _ the 
therapists on the check list, for the whole 
sample, therefore, cannot be attributed 
to the presence of any group of cases in 
the sample which were not validly inter- 
preted as whole reports. 
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\. THe Cueck List APPROACH 


HE CHECK LIST approach was em- 
"T propel in this study for two purposes: 
(a) to study the possibilities of validating 
a projective technique on a set of isolated 
interpretative statements, and (b) to de- 
termine whether the behavior of the 
individual on the test and in a life situa- 
tion could be described by the same 
statement. The results of validating an 
accepted projective technique, the Ror- 
schach, on this check list indicate that 
this approach is not applicable to the 
validation of such projective techniques. 
In view of the fact that both the Ror- 
schach interpreters and the therapists 
used this check list reliably when de- 
scribing their respective observations, the 
main conclusion is that the check list 
approach may be applicable to the study 
of personality descriptions only when a 
common set of concepts is maintained as 
a reference point for inferring the total 
pattern of the individual’s functioning. 

In general, these findings support the 
contentions of Vernon (19), Frank (3), 
and other investigators who have argued 
that a description of the interrelated 
functioning of an individual can be vali- 
dated only as a whole. In particular, it 
has been demonstrated that a total and 
integrated picture of .the individual's 
personality may be valid, even though 
there may be no more than chance agree- 
ment between the judgments of a test 
interpreter and a criterion in regard to 
isolated statements about the individual's 
functioning. 

This check list approach was a more 
rigid and exacting test of the validity of 
separate interpretative statements than 
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the “item analysis” design employed by 
Harrison (4) and by Cronbach (2). In the 
first place, neither of these investigators 
attempted to test whether the behavior 
of the individual in both the test situa- 
tion and in the life situation could be 
described by a common set of statements. 
Secondly, in this check list approach, the 
isolated statements were selected accord- 
ing to a definite rationale, i.e., the scheme 
of “area” and “dimensions” of personal- 
ity. 

The results of the present study indi- 
cate that the behavior of the individual in 
both the test situation and a life situa- 
tion could not be satisfactorily described 
by the same statement. Since the “item 
analysis” method does not make this 
demand on the test, it is probably a 
sounder approach. However, in the use 
of an “item analysis” approach it is rec- 
ommended that the rationale for select- 
ing statements from the whole reports 
be clearly stated. 


* B. THe MATCHING APPROACH 


The chief advantage of the matching 
approach, according to Vernon (18), is 
that for whole interpretative reports, this 
approach tests the validity of the most 
essential features of the interpretation of 
projective techniques, i.e., the accuracy 
with which the interrelated pattern of 
the individual’s functioning is described. 
The findings of the present study support 
Vernon’s contention. By means of the 
matching approach, validity was demon- 
strated for descriptions of personality 
which emphasized these interrelation- 
ships. On the other hand, no validity 
was obtained when isolated interpretative 
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statements were applied in which the 
relationships between various function- 
ings of the personality were not elabo- 
rated. 

One of the major concerns in the 
present study of the matching approach 
was the effect of the heterogeneity of the 
matching groups. This study indicated 
that use of matching groups of an “opti- 
mum” heterogeneity resulted in a smaller 
number of successful matchings than 
were obtained in previous studies using 
randomly selected groups. Since the 
matching approach is essentially a test 
of interindividual differentiation, care- 
ful attention must be paid to the nature 
of the sample of individuals from whom 
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a particular individual is to be differenti. 
ated. 

The chief criticism directed againy 
the matching approach has been that jt 
does not provide a test of the accuracy 
with which various part-functionings 
within this whole pattern are delineated, 
The present study attempted, without 
success, to test this intraindividual dil. 
ferentiation of an interpretation, by 
means of the check list approach. Since 
the completion of the present study, an 
“item analysis” design has been suggested 
by Cronbach (2) which, if used in con- 
junction with a matching approach, may 
provide a thorough statistical method for 
the validation of projective techniques, 
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